Introducing SpeedrunBench: A Challenging Benchmark for Frontier Agents

Today we introduce SpeedrunBench, a benchmark of 10 retro and open-source titles spanning multiple genres that scores LLM agents on the number of frames needed to reach a goal state. Most game benchmarks measure “can the agent complete the game”? That metric saturates the moment frontier models complete the game. Human speedrunning communities (https://www.speedrun.com/https://tasvideos.org/) have spent decades pushing the same games well past completion by discovering new routes and techniques, which makes frame count to the milestone of a game a natural objective.

‍

‍
SpeedrunBench evaluates agents under three settings. In ONLINE, the setting prior work adopts, the agent sees the current frame and predicts the next action. In OFFLINE, the agent edits a complete frame-indexed action trace, replays it, and reads back the frame at which the goal was reached, repeating over a fixed turn budget. We split this into OFFLINE-SEED, where the agent starts from a working reference trajectory, and OFFLINE-SCRATCH, where it has to construct one itself. Each setting isolates a different capability: multimodal understanding, exploration, and route discovery, with long-horizon coherence demanded by all three. The suite also pairs documented commercial titles with open-source counterparts of the same genre, so contamination from pretraining can be tested.

‍

Speedrunning separates models that completion alone does not: 1) on Pokémon Blue, every agent that reaches the first badge is far off from the human world record, with the best runs roughly 4x slower than the ~40,435-frame (11 mins 17 secs) human record and the worst nearly 12x. 2) Seeding a single mediocre run and optimize from it hurts more than it helps for frontier models: on the first stage of SuperTux, all agents that could clear the level unaided posted lower frame counts from scratch than when seeded, by up to 5.1x. We read this as agents inheriting a slow route and routing freely when not. But the seed is what makes the task tractable for weaker agents since three models failed to reach the goal from scratch at all and many models cannot reach the goal on more complicated games like Pokémon. Finally, the number of evaluations or runs within an iteration matters more than model choice: with up to 200 runs per iteration, GLM 5.2 is 58 frames ~0.97 seconds away from the human world record on Super Mario Land 1-1. 

Paper: https://cdn.patronus.ai/SpeedrunBench.pdf

Benchmark: https://huggingface.co/datasets/PatronusAI/speedrunbench

Game: https://speedrunbench.com/

‍